Core group memory processing with group b-float encoding

ABSTRACT

A memory processing unit (MPU) can include a first memory, a second memory, a plurality of processing regions and control logic. The first memory can include a plurality of memory regions. The plurality of memory regions can be organized in a plurality of memory blocks. The plurality of memory regions can be configured to store integer, B-float, and/or Group B-float encode data. The plurality of processing regions can be interleaved between the plurality of processing regions of the first memory. The plurality of processing regions can be organized in a plurality of core groups include a plurality of compute cores. The compute groups in the processing regions can be coupled to a plurality of adjacent memory blocks in the adjacent memory regions. The second memory can be coupled to the plurality of processing regions.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 63/310,031 filed Feb. 14, 2022, which is incorporatedherein in its entirety.

BACKGROUND OF THE INVENTION

Computing systems have made significant contributions toward theadvancement of modern society and are utilized in a number ofapplications to achieve advantageous results. Applications such asartificial intelligence, machine learning, big data analytics and thelike perform computations on large amounts of data. In conventionalcomputing systems, data is transferred from memory to one or moreprocessing units, the processing units perform calculations on the data,and the results are then transferred back to memory. The transfer oflarge amounts of data from memory to the processing unit and back tomemory takes time and consumes power. Accordingly, there is a continuingneed for improved computing systems that reduce processing latency, datalatency and or power consumption.

SUMMARY OF THE INVENTION

The present technology may best be understood by referring to thefollowing description and accompanying drawings that are used toillustrate embodiments of the present technology directed toward memoryprocessing architectures.

In one embodiment, a memory processing unit (MPU) can include a firstmemory and a plurality of processing regions. The first memory caninclude a plurality of memory regions, wherein the plurality of memoryregions can be configured in a corresponding pluralities of memoryblocks. The memory blocks can be configured to store Brian FloatingPoint (B-float) encoded data and or group B-float encoded data Theplurality of processing regions can be interleaved between the pluralityof regions of the first memory, wherein the processing regions include aplurality of core groups, and wherein the core groups include one ormore compute cores.

In another embodiment, a memory processing unit (MPU) can include afirst memory and a plurality of processing regions interleaved between aplurality of regions of the first memory. The plurality of memoryregions can be configured in corresponding pluralities of memory blocksand the plurality of processing regions can be configure incorresponding pluralities of core groups. The plurality of core groupsof respective ones of the plurality of processing regions can be coupledbetween adjacent ones of the plurality of memory regions of the firstmemory. The memory blocks are configured to store Group B-float encodedfeature map pixels.

In another embodiment, a memory processing method can includeconfiguring a first memory to store Group B-float encoded data, whereinthe first memory includes a plurality of regions. Data flow betweencompute cores of one or more of a plurality of processing regions andcorresponding adjacent ones of the plurality of regions of the firstmemory can be configured. Data flow between a second memory and thecompute cores of the one or more of the plurality of processing regionscan also be configured. Data flow can also be configured between computecores within respective ones of the one or more of the plurality ofprocessing regions. One or more sets of compute cores of one or more ofthe plurality of processing regions can be configured to performrespective compute functions of a neural network model. Weights for theneural network model can be loaded into the second memory, andactivation data for the neural network model can be loaded into one ormore of the plurality of regions of the first memory. Data movementbetween one or more compute cores producing given data and one or moreother compute cores consuming the given data can be synchronized basedon the neural network model.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present technology are illustrated by way of exampleand not by way of limitation, in the figures of the accompanyingdrawings and in which like reference numerals refer to similar elementsand in which:

FIG. 1 shows a memory processing unit (MPU), in accordance with aspectsof the present technology.

FIG. 2 shows a memory processing unit (MPU), in accordance with aspectsof the present technology.

FIG. 3 shows a memory processing unit (MPU), in accordance with aspectsof the present technology.

FIG. 4 shows a memory processing unit (MPU), in accordance with aspectsof the present technology.

FIG. 5 shows a memory processing unit (MPU), in accordance with aspectsof the present technology.

FIG. 6 illustrates an exemplary mapping of a neural network to computecores, in accordance with aspects of the present technology.

FIG. 7 illustrates an exemplary compute core mapping, in accordance withaspects of the present technology.

FIGS. 8A-8B show an exemplary computation of multiple output feature mappixels, in accordance with aspects of the present technology.

FIG. 9 shows configuration of dataflows in a memory processing unit(MPU), in accordance with aspects of the present technology.

FIG. 10 shows a near memory (M) compute core, in accordance with aspectsof the present technology.

FIG. 11 shows an arithmetic (A) compute core, in accordance with aspectsof the present technology.

FIG. 12 shows an input (I) core, in accordance with aspects of thepresent technology.

FIG. 13 shows an output (O) core, in accordance with aspects of thepresent technology.

FIGS. 14-17 illustrate a whole channel compute core configuration, inaccordance with aspects of the present technology.

FIGS. 18-21 show a second memory region polymorphic compute coreconfiguration, in accordance with aspects of the present technology.

FIGS. 22-25 show a first memory region polymorphic compute coreconfiguration, in accordance with aspects of the present technology.

FIGS. 26-29 show a compound compute core configuration, in accordancewith aspects of the present technology.

FIG. 30 shows a first memory region sharing feature of the memoryprocessing unit (MPU), in accordance with aspects of the presenttechnology.

FIGS. 31A and 31B illustrate an exemplary buffer utilization by aconsumer and a producer, in accordance with aspects of the presenttechnology.

FIGS. 32A-32D illustrate an exemplary shared partial buffer for a 3×3kernel size, in accordance with aspects of the present technology.

FIGS. 33A and 33B illustrate an exemplary shared partial buffer for a3×3 kernel size with a 2×2 stride, in accordance with aspects of thepresent technology.

FIG. 34 illustrates an example branching dataflow utilizing a fullfeature-map buffer, in accordance with aspects of the presenttechnology.

FIG. 35 illustrates an exemplary branching dataflow utilizing a partialfeature-map buffer, in accordance aspects of the present technology.

FIG. 36 illustrates an exemplary branching dataflow utilizing a partialfeature-map buffer, in accordance aspects of the present technology.

FIG. 37 shows a memory processing unit (MPU), in accordance with aspectsof the present technology.

FIG. 38 shows an inter-layer-communication method, in accordance withaspect of the present technology.

FIG. 39 shows respective shared buffers and corresponding respective ILCentry indexes, in accordance with aspects of the present technology.

FIG. 40 illustrates tracking of access to a shared respective buffer ina respective ILC entry index, in accordance with aspects of the presenttechnology.

FIG. 41 illustrates a 4-dimension array, in accordance with aspects ofthe present technology.

FIG. 42 illustrates a 3-dimension array, in accordance with aspects ofthe present technology.

FIG. 43 illustrates a 2-dimension array, in accordance with aspects ofthe present technology.

FIG. 44 shows a memory macro of a memory processing unit (MPU), inaccordance with aspects of the present technology.

FIG. 45 shows a method of fitting arrays into a 2-dimension memory, inaccordance with aspects of the present technology.

FIG. 46 illustrates expansion of a 3-dimension array, in accordance withaspects of the present technology.

FIG. 47 illustrates expansion of a 2-dimension array, in accordance withaspects of the present technology.

FIG. 48 illustrates quantization of an array, in accordance with aspectsof the present technology.

FIG. 49 illustrates flattening of a quantized array, in accordance withaspects of the present technology.

FIG. 50 illustrates reshaping of a flattened array, in accordance withaspects of the present technology.

FIG. 51 illustrates rotating of a reshaped array, in accordance withaspects of the present technology.

FIG. 52 illustrates loading virtual channels of the reshaped array intophysical channels of memory, in accordance with aspects of the presenttechnology.

FIGS. 53A-53D illustrate fetching from a wide memory block, inaccordance with aspects of the present technology.

FIGS. 54A-54C illustrate a write back to a wise memory block, inaccordance with aspects of the present technology.

FIG. 55 illustrates a reshape function, in accordance with aspects ofthe present technology.

FIGS. 56A-56D illustrate a deconvolution function, in accordance withaspects of the present technology.

FIG. 57 illustrates a deconvolution function, in accordance with aspectsof the present technology.

FIG. 58 illustrates a sigmoid function, in accordance with aspects ofthe present technology.

FIG. 59 illustrates a feature map, in accordance with aspects of thepresent technology.

FIG. 60 illustrates a B-float encoded data value, in accordance withaspects of the present technology.

FIG. 61 illustrates a logical view of a feature map encoded in GroupB-float from an output side of a computer core, in accordance withaspects of the present technology.

FIG. 62 illustrates a logical view of a feature map encoded in GroupB-float from an input side of a compute core, in accordance with aspectsof the present technology.

FIG. 63 illustrates another logical view of a feature map and weightsencoded in Group B-float from an input side of a compute core, inaccordance with aspects of the present technology.

FIG. 64 illustrates storage of B-float encoded feature map data in anarrow flat memory organization, in accordance with aspects of thepresent technology.

FIG. 65 illustrates storage of B-float encoded feature map data in awide memory organization, in accordance with aspects of the presenttechnology.

FIG. 66 illustrates storage of Group B-float encoded feature map data ina wide memory organization, in accordance with aspects of the presenttechnology.

FIG. 67 illustrates accuracy of calculations on Group B-float encodedResNet-50 feature map pixels values for different group sizes, inaccordance with aspects of the present technology.

FIG. 68 illustrates accuracy of calculations on Group B-float MobileNetfeature map pixels values for different group sizes, in accordance withaspects of the present technology.

DETAILED DESCRIPTION OF THE INVENTION

Reference will now be made in detail to the embodiments of the presenttechnology, examples of which are illustrated in the accompanyingdrawings. While the present technology will be described in conjunctionwith these embodiments, it will be understood that they are not intendedto limit the technology to these embodiments. On the contrary, theinvention is intended to cover alternatives, modifications andequivalents, which may be included within the scope of the invention asdefined by the appended claims. Furthermore, in the following detaileddescription of the present technology, numerous specific details are setforth in order to provide a thorough understanding of the presenttechnology. However, it is understood that the present technology may bepracticed without these specific details. In other instances, well-knownmethods, procedures, components, and circuits have not been described indetail as not to unnecessarily obscure aspects of the presenttechnology.

Some embodiments of the present technology which follow are presented interms of routines, modules, logic blocks, and other symbolicrepresentations of operations on data within one or more electronicdevices. The descriptions and representations are the means used bythose skilled in the art to most effectively convey the substance oftheir work to others skilled in the art. A routine, module, logic blockand/or the like, is herein, and generally, conceived to be aself-consistent sequence of processes or instructions leading to adesired result. The processes are those including physical manipulationsof physical quantities. Usually, though not necessarily, these physicalmanipulations take the form of electric or magnetic signals capable ofbeing stored, transferred, compared and otherwise manipulated in anelectronic device. For reasons of convenience, and with reference tocommon usage, these signals are referred to as data, bits, values,elements, symbols, characters, terms, numbers, strings, and/or the likewith reference to embodiments of the present technology.

It should be borne in mind, however, that these terms are to beinterpreted as referencing physical manipulations and quantities and aremerely convenient labels and are to be interpreted further in view ofterms commonly used in the art. Unless specifically stated otherwise asapparent from the following discussion, it is understood that throughdiscussions of the present technology, discussions utilizing the termssuch as “receiving,” and/or the like, refer to the actions and processesof an electronic device such as an electronic computing device thatmanipulates and transforms data. The data is represented as physical(e.g., electronic) quantities within the electronic device's logiccircuits, registers, memories and/or the like, and is transformed intoother data similarly represented as physical quantities within theelectronic device.

In this application, the use of the disjunctive is intended to includethe conjunctive. The use of definite or indefinite articles is notintended to indicate cardinality. In particular, a reference to “the”object or “a” object is intended to denote also one of a possibleplurality of such objects. The use of the terms “comprises,”“comprising,” “includes,” “including” and the like specify the presenceof stated elements, but do not preclude the presence or addition of oneor more other elements and or groups thereof. It is also to beunderstood that although the terms first, second, etc. may be usedherein to describe various elements, such elements should not be limitedby these terms. These terms are used herein to distinguish one elementfrom another. For example, a first element could be termed a secondelement, and similarly a second element could be termed a first element,without departing from the scope of embodiments. It is also to beunderstood that when an element is referred to as being “coupled” toanother element, it may be directly or indirectly connected to the otherelement, or an intervening element may be present. In contrast, when anelement is referred to as being “directly connected” to another element,there are not intervening elements present. It is also to be understoodthat the term “and or” includes any and all combinations of one or moreof the associated elements. It is also to be understood that thephraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting.

Referring now to FIG. 1 , a memory processing unit, in accordance withaspects of the present technology, is shown. The memory processing unit100 can include a plurality of memory regions 110-130, a plurality ofprocessing regions 135-150, one or more communication links 155, and oneor more centralized or distributed control circuitry 160. The pluralityof memory regions 110-130 can also be referred to as activation memory.The plurality of processing regions 135-150 can be interleaved betweenthe plurality of memory regions 110-130. The processing regions 135-150can be interleaved in an alternating regular pattern of a processingregion 135, a memory region 115, a processing region 140, a memoryregion 120, a processing region 145, and so on. In one implementation,the plurality of memory regions 110-130 and the plurality of processingregions 135-150 can have respective predetermine sizes. The plurality ofprocessing regions 135-150 can have the same design. Similarly, theplurality of memory region 110-130 can also have the same design. In oneimplementation, the plurality of memory regions 110-130 can be staticrandom access memory (SRAM), and the plurality of processing regions135-150 can include one or more arrays of resistive random access memory(ReRAM), magnetic random access memory (MRAM), phase change randomaccess memory (PCRAM), Flash memory (FLASH), or the like.

One or more of the plurality of processing regions 135-150 can beconfigured to perform one or more computation functions, one or moreinstances of one or more computation functions, one or more segments ofone or more computation functions, or the like. For example, a firstprocessing region 135 can be configured to perform two computationfunctions, and a second processing region 140 can be configured toperform a third computation function. In another example, the firstprocessing region 135 can be configured to perform three instances of afirst computation function, and the second processing region 140 can beconfigured to perform a second and third computation function. In yetanother example, a given computation function can have a size largerthan the predetermined size of the one or more processing regions. Insuch case, the given computation function can be segmented, and thecomputation function can be configured to be performed on one or more ofthe plurality of processing units 135-150. The processing regions135-150 can each include one or more memory processing units, memoryprocessing unit cores, or the like. The memory processing units and orcores can implement computation functions in arrays of memory cellswithout changing the basic memory array structure. The one or morecentralized or distributed control circuitry 160 can configure the oneor more computation functions of the one or more of the plurality ofprocessing regions 135-150. The computation functions can include, butare not limited to, vector products, matrix-dot-products, convolutions,min/max pooling, averaging, scaling, and or the like.

A central data flow direction can be utilized with the plurality ofmemory regions 110-130 and plurality of processing regions 135-150. Theone or more centralized or distributed control circuitry 160 can controldata flow into each given one of the plurality of processing regions135-150 from a first adjacent one of the plurality of memory regions110-130 to a second adjacent one of the plurality of memory regions110-130. For example, the one or more control circuitry 160 canconfigure data to flow into a first processing region 135 from a firstmemory region 110 and out to a second memory region 115. Similarly, theone or more control circuitry 160 can configure data to flow into asecond processing region 140 from the second memory region 115 and outto a third memory region 120. The control circuitry 160 can include acentralized control circuitry, distributed control circuitry or acombination thereof. If distributed, the control circuitry 160 can belocal to the plurality of memory regions 110-130, the plurality ofprocessing regions 135-150, and or one or more communication links 155.

In one implementation, the plurality of memory regions 110-130 and theplurality of processing regions 135-150 can be columnal interleaved witheach other. The data can be configured by the one or more centralized ordistributed control circuitry 160 to flow between adjacent columnalinterleaved processing regions 135-150 and memory regions 110-130 in across-columnal direction. In one implementation, the data can flow in aunidirectional cross-columnal direction between adjacent processingregions 135-150 and memory regions 110-130. For example, data can beconfigured to flow from a first memory region 110 into a firstprocessing region 135, from the first processing region 135 out to asecond memory region 115, from the second memory region 115 into asecond processing region 140, and so on. In another implementation, thedata can flow in a bidirectional cross-columnal direction betweenadjacent processing regions 135-150 and memory regions 110-130. Inaddition or alternatively, data within respective ones of the processingregion 135-150 can flow between functions within the same processingregion. For example, for a first processing region 135 configured toperform two computation functions, data can flow from the firstcomputation function directly to the second computation function withoutbeing written or read from an adjacent memory region.

The one or more communication links 155 can be coupled between theinterleaved plurality of memory region 110-130 and plurality ofprocessing regions 135-150. The one or more communication links 155 canbe configured for moving data between non-adjacent ones of the pluralityof memory regions 110-130, between non-adjacent ones of the plurality ofprocessing regions 135-150, or between non-adjacent ones of a givenmemory region and a given processing region. For example, the one ormore communication links 155 can be configured for moving data betweenthe second memory region 115 and a fourth memory region 125. In anotherexample, the one or more communication links 155 can be configured formoving data between the first processing region 135 and a thirdprocessing region 145. In another example, the one or more communicationlinks 155 can be configured for moving data between the second memoryregion 115 and the third processing region 145, or between the secondprocessing unit 140 and a fourth memory region 125.

Generally, the plurality of memory regions 110-130 and the plurality ofprocessing regions 135-150 are configured such that partial sums move ina given direction through a given processing region. In addition, theplurality of memory regions 110-130 and the plurality of processingregions 135-150 are generally configured such that edge outputs move ina given direction from a given processing region to an adjacent memoryregion. The terms partial sums and edge outputs are used herein to referto the results of a given computation function or a segment of acomputation function. The computation functions of the plurality ofprocessing regions 135-150 and the dataflow between the plurality ofprocessing regions 135-150 and the memory regions 110-130 can beconceptualized as a plurality of produces and consumers. Computationfunctions of a given processing region can consume data form a givenmemory region and produce output data to a next memory region. Theoutput data stored in the given memory region can then be consumed bycomputation functions of a next given processing region. Accordingly,producers and consumers communicate through shared memory regions110-130. The computation functions and dataflow between adjacentprocessing regions 135-150 and memory regions 110-130 can be mapped toensure adjacency requirements are met. The shared data can therefore besynchronized in a dataflow manner without a global centralized controlunit.

Referring to FIG. 2 , a memory processing unit (MPU), in accordance withaspects of the present technology, is shown. The memory processing unit200 can include a first memory 202-210 and a plurality of processingregions 212-216. The first memory 202-210 can include a plurality ofmemory regions. The plurality of processing regions 212-216 can beinterleaved between the plurality of regions 202-210 of the firstmemory. The processing regions 212-216 and plurality of first memoryregions 202-210 can be interleaved in an alternating regular pattern ofa processing region 212, a memory region 204, a processing region 214, amemory region 206, a processing region, and so on. The plurality offirst memory regions 202-210 can be volatile memory, such as staticrandom-access memory (SRAM) or the like. The processing regions 212-216can include a plurality of compute cores 220-232. The plurality ofcompute cores 220-232 of respective ones of the plurality of processingregions 212-216 can be coupled between adjacent ones of the plurality ofregions of the first memory 202-210. For example, the compute cores220-228 of a first processing region 212 can be coupled between a firstregion 202 and a second region 204 of the first memory region 202-210.The compute cores 220-232 in each respective processing region 212-216can be configurable in one or more clusters 234-238. For example, afirst set of compute cores 220, 222 in a first processing region 212 canbe configurable in a first cluster 234. Similarly, a second set ofcompute cores 224-228 in the first processing region can be configurablein a second cluster 236. The plurality of compute cores 220-232 ofrespective ones of the plurality of processing regions 212-216 can alsobe configurably couplable in series. For example, a set of compute cores220-224 in a first processing region 212 can be communicatively coupledin series, with a second compute core 222 receiving data and orinstructions from a first compute core 220, and a third compute core 224receiving data and or instructions from the second compute core 222.

The memory processing unit 200 can also include a second memory 218. Thesecond memory 218 can be coupled to the plurality of processing regions212-216. The second memory 218 can optionally be logically or physicallyorganized into a plurality of regions. The plurality of regions of thesecond memory 218 can be associated with corresponding ones of theplurality of processing region 212-216. In addition, the plurality ofregions of the second memory 218 can include a plurality of blocksorganized in one or more macros. The second memory can be non-volatilememory, such as resistive random-access memory (RRAM), magneticrandom-access memory (MRAM), flash memory (FLASH) or the like. Thesecond memory can alternatively be volatile memory.

One or more of the plurality of processing regions 212-216 can beconfigured to perform one or more computation functions, one or moreinstances of one or more computation functions, one or more segments ofone or more computation functions, or the like. For example, a firstprocessing region 212 can be configured to perform two computationfunctions, and a second processing region 214 can be configured toperform a third computation function. In another example, the firstprocessing region 212 can be configured to perform three instances of afirst computation function, and the second processing region 214 can beconfigured to perform a second and third computation function.Similarly, the compute cores 220-232 can be configured to perform one ormore computation functions, one or more instances of one or morecomputation functions, one or more segment of one or more computationfunctions, or the like. The compute cores 220-232 of the plurality ofprocessing regions 212-216 can each include one or more memoryprocessing units, memory processing unit cores, or the like. The memoryprocessing units and or cores can implement computation functions inarrays of memory cells without changing the basic memory arraystructure.

The memory processing unit 200 can further include aninter-layer-communication (ILC) unit 240. The ILC unit 240 can be globalor distributed across the plurality of processing regions 212-216. Inone implementation, the ILC unit 240 can include a plurality of ILCmodules 242-246, wherein each ILC module can be coupled to a respectiveprocessing region 212-216. Each ILC module can also be coupled to therespective regions of the first memory 202-210 adjacent thecorresponding respective processing regions 212-216. Theinter-layer-communication unit 240 can be configured to synchronize datamovement between one or more compute cores producing given data and oneor more other compute cores consuming the given data.

The memory processing unit 200 can further include one or moreinput/output stages 242, 244. The one or more input/output stages 242,244 can be coupled to one or more respective regions of the first memory202-210. The one or more input/output stages 242, 244 can include one ormore input ports, one or more output ports, and or one or moreinput/output ports. The one or more input/output stages 242, 244 can beconfigured to stream data into or out of the memory processing unit 200.For example, one or more of the input/output (I/O) stages can beconfigured to stream data into a first one of the plurality of regionsof the first memory 202-210. Similarly, one or more input/output (I/O)stages can be configured to stream data out of a last one of theplurality of regions of the first memory 202-210.

The plurality of processing regions 212-216 can be configurable formemory-to-core dataflow from respective ones of the plurality of regionsof the first memory 202-210 to one or more cores 220-232 within adjacentones of the plurality of processing regions 212-216. The plurality ofprocessing regions 212-216 can also be configurable for core-to-memorydataflow from one or more cores 220-232 within ones of the plurality ofprocessing regions 212-216 to adjacent ones of the plurality of regionsof the first memory 202-210. The plurality of processing regions 212-216and plurality of regions of the first memory 202-210 can also beconfigured for memory-to-core-to-memory data flow. For example, thedataflow can be configured for a given direction from given ones of theplurality of regions of the first memory 202-210 through respective onesof the plurality of processing regions to adjacent ones of the pluralityof regions of the first memory 202-210. In one implementation, thecomputation functions of compute cores and dataflow between processingregions 212-216 and first memory 202-210 can be organized to ensureadjacency requirements so that dataflow of shared data can besynchronized therebetween without a global centralized control unit.

The plurality of processing regions 212-216 can also be configurable formemory-to-core data flow from the second memory 218 to one or more cores220-232 of corresponding ones of the plurality of processing regions212-216. If the second memory 218 is logically or physically organizedin a plurality of regions, respective ones of the plurality of regionsof the second memory 218 can be configurably couplable to one or morecompute cores in respective ones of the plurality of processing regions212-216.

The plurality of processing regions 212-216 can be further configurablefor core-to-core data flow between select adjacent compute cores 220-232in respective ones of the plurality of processing regions 212-216. Forexample, a given core 224 can be configured to share data, accessed froman adjacent portion of the first memory 202, with one or more othercores 226-228 configurably coupled in series with the given compute core224. In another example, a given core 220 can be configured to pass dataaccessed from the second memory 218 with one or more other cores 222configurably coupled in series with the given compute core 220. In yetanother example, a given compute core 220 can pass a result, such as apartial sum, computed by the given compute core 220, to one or moreother cores 222 configurably coupled in series with the given computecore 220.

Referring to FIG. 3 , a memory processing unit (MPU), in accordance withaspects of the present technology, is shown. The memory processing unit300 can include a first memory and a plurality of processing regions312-316. The first memory can include including a plurality of regions302-310. The plurality of processing regions 312-316 can be interleavedbetween the plurality of regions of the first memory 302-310. Theprocessing regions 312-316 can include a plurality of compute cores320-332. The plurality of compute cores 320-332 of respective ones ofthe plurality of processing regions 312-316 can be coupled betweenadjacent ones of the plurality of regions of the first memory 302-310.For example, the compute cores 320-328 of a first processing region 312can be coupled between a first region 302 and a second region 304 of thefirst memory 302-310. The compute cores 320-332 in each respectiveprocessing region 312-316 can be configurable in one or more clusters334-338. For example, a first set of compute cores 320, 322 in a firstprocessing region 312 can be configurable in a first cluster 334.Similarly, a second set of compute cores 324-328 in the first processingregion can be configurable in a second cluster 336. The plurality ofcompute cores 320-332 of respective ones of the plurality of processingregions 312-316 can also be configurably couplable in series. Forexample, a set of compute cores 320-324 in a first processing region 312can be communicatively coupled in series, wherein a second compute core322 receiving data and or instructions from a first compute core 320,and a third compute core 324 receiving data and or instructions from thesecond compute core 322.

The memory processing unit 300 can also include a second memory 318. Thesecond memory 318 can be coupled to the plurality of processing regions312-316. The second memory 318 can optionally be logically or physicallyorganized into a plurality of regions. The plurality of regions of thesecond memory 318 can be associated with corresponding ones of theplurality of processing region 312-316. In addition, the plurality ofregions of the second memory 318 can include a plurality of blocksorganized in one or more macros. The first memory 302-310 can bevolatile memory, such as static random-access memory (SRAM) or the like.The second memory can be non-volatile memory, such as resistiverandom-access memory (RRAM), magnetic random-access memory (MRAM), flashmemory (FLASH) or the like. The second memory can alternatively bevolatile memory.

The memory processing unit 300 can further include aninter-layer-communication (ILC) unit 340. The ILC unit 340 can be globalor distributed across the plurality of processing regions 312-316. Inone implementation, the ILC unit 340 can include a plurality of ILCmodules, wherein each ILC module can be coupled to a respectiveprocessing regions 312-316. Each ILC module can also be coupled to therespective regions of the first memory 302-310 adjacent thecorresponding respective processing regions 312-316. Theinter-layer-communication unit 340 can be configured to synchronize datamovement between one or more compute cores producing given data and oneor more other compute cores consuming the given data. The inter-layercommunication unit 340 can map the computations functions of computecores and dataflow between processing regions 312-316 and first memory302-310 on an adjacency basis so that dataflow of shared data can besynchronized therebetween.

The memory processing unit 300 can further include one or moreinput/output stages 348, 350. The one or more input/output stages 348,350 can be coupled to one or more respective regions of the first memory302-310. The one or more input/output stages 348, 350 can include one ormore input ports, one or more output ports, and or one or moreinput/output ports. The one or more input/output stages 348, 350 can beconfigured to stream data into or out of the memory processing unit 300.For example, one or more of the input/output (I/O) stages can beconfigured to stream data into a first one of the plurality of regionsof the first memory 302-310. Similarly, one or more input/output (I/O)stages can be configured to stream data out of a last one of theplurality of regions of the first memory 302-310.

The plurality of processing regions 312-316 can be configurable formemory-to-core dataflow from respective ones of the plurality of regionsof the first memory 302-310 to one or more cores 320-332 within adjacentones of the plurality of processing regions 312-316. The plurality ofprocessing regions 312-316 can also be configurable for core-to-memorydataflow from one or more cores 320-332 within ones of the plurality ofprocessing regions 312-316 to adjacent ones of the plurality of regionsof the first memory 302-310. In one implementation, the dataflow can beconfigured for a given direction from given ones of the plurality ofregions of the first memory 302-310 through respective ones of theplurality of processing regions to adjacent ones of the plurality ofregions of the first memory 302-310.

The plurality of processing regions 312-316 can also be configurable formemory-to-core data flow from the second memory 318 to one or more cores320-332 of corresponding ones of the plurality of processing regions312-316. If the second memory 318 is logically or physically organizedin a plurality of regions, respective ones of the plurality of regionsof the second memory 318 can be configurably couplable to one or morecompute cores in respective ones of the plurality of processing regions312-316.

The plurality of processing regions 312-316 can be further configurablefor core-to-core data flow between select adjacent compute cores 320-332in respective ones of the plurality of processing regions 312-316. Forexample, a given core 324 can be configured to share data, accessed froman adjacent portion of the first memory 302, with one or more othercores 326-328 configurably coupled in series with the given compute core324. In another example, a given core 320 can be configured to passdata, accessed from the second memory 318, with one or more other cores322 configurably coupled in series with the given compute core 320. Inyet another example, a given compute core 320 can pass a result, such asa partial sum, computed by the given compute core 320, to one or moreother cores 322 configurably coupled in series with the given computecore 320.

The plurality of processing regions 312-316 can include one or more nearmemory (M) compute cores. The one or more near memory (M) compute corescan be configurable to compute neural network functions. For example,the one or more near memory (M) compute cores can be configured tocompute vector-vector products, vector-matrix products, matrix-matrixproducts, and the like, and or partial products thereof.

The plurality of processing regions 312-316 can also include one or morearithmetic (A) compute cores. The one or more arithmetic (A) computecores can be configurable to compute arithmetic operations. For example,the arithmetic (A) compute cores can be configured to compute mergeoperations, arithmetic calculations that are not supported by the nearmemory (M) compute cores, and or the like.

The plurality of input and output regions 348, 350 can also include oneor more input/output (I/O) cores. The one or more input/output (I/O)cores can be configured to access input and or output ports of thememory processing unit (MPU) 300. The term input/output (I/O) core asused herein can refer to cores configured to access input ports, coresconfigured to access output ports, or cores configured to access bothinput and output ports.

The compute cores 320-332 can include a plurality of physical channelsconfigurable to perform computations, accesses and the like,simultaneously with other cores within respective processing regions312-316, and or simultaneously with other cores in other processingregions 312-316. The compute cores 320-332 of respective ones of theplurality of processing regions 312-316 can be associated with one ormore blocks of the second memory 318. The compute cores 320-332 ofrespective ones of the plurality of processing regions 312-316 can beassociated with respective slices of the second plurality of memoryregions. The cores 320-332 can also include a plurality of configurablevirtual channels.

Referring now to FIG. 4 , a memory processing unit, in accordance withaspects of the present technology, is shown. The memory processing unit400 can include a first memory region and a plurality of processingregion 410-414. The first memory can include a plurality of memoryregions 402-408. The plurality of processing regions 410-414 can beinterleaved between the plurality of memory regions 402-408 of the firstmemory. In one implementation, the plurality of memory regions 402-408and the plurality of processing regions 410-414 can have respectivepredetermine sizes. One or more of the plurality of memory regions402-408 can include a plurality of memory blocks 416-432. One or moreprocessing regions 410-414 can also include a plurality of core groups434-448. A core group 434-448 can include one or more computer cores.The computer cores in a respective core group can be arranged in one ormore compute clusters. One or more of the plurality of core groups of arespective one of the plurality of processing regions can be coupledbetween adjacent ones of the plurality of memory regions of the firstmemory. In one implementation, a given core group can be coupled to aset of directly adjacent memory blocks, while not coupled to the othermemory blocks of the adjacent memory regions. In other words, a coregroup of a respective processing region can be coupled to a set ofmemory blocks that are proximate to the given core group, while notcoupled to memory blocks in the adjacent memory regions that are distalfrom the given core group. For example, a first core group 434 of afirst processor region 410 can be coupled between a first memory block416 of a first memory region 402 and a first memory block 422 of asecond memory region 404. A second core group 436 of the first processorregion 410 can be coupled to the first and a second memory block 416,418 of the first memory region 402 and the first and a second memoryblock 422, 424 of the second memory region 404. The second core group436 of the first processor region 410 can also be coupled between thefirst and a third core groups 434, 438 of the first processor region410.

One or more of the compute cores, and or one or more core groups of theplurality of processing regions 410-414 can be configured to perform oneor more computation functions, one or more instances of one or morecomputation functions, one or more segments of one or more computationfunctions, or the like. For example, a first computer core, a first coregroup 434 or a first processing region 410 can be configured to performtwo computation functions, and a second computer core, second core groupor second processing region 412 can be configured to perform a thirdcomputation function. In another example, a first compute core, thefirst core group 434 or the first processing region 410 can beconfigured to perform three instances of a first computation function,and a second compute core, second core group or second processing region412 can be configured to perform a second and third computationfunction. In yet another example, a given computation function can havea size larger than the predetermined size of a compute core, core groupor one or more processing regions. In such case, the given computationfunction can be segmented, and the computation function can beconfigured to be performed on one or more compute cores, one or morecore groups or one or more of the processing regions 410-414. Thecomputation functions can include, but are not limited to, vectorproducts, matrix-dot-products, convolutions, min/max pooling, averaging,scaling, and or the like.

The memory processing unit 400 can also include one or more inter-layercommunication (ILC) units 450-456. The ILC unit 450-456 can be global ordistributed across the plurality of processing regions 410-414. In oneimplementation, the ILC unit 450-456 can include a plurality of ILCmodules 450-456, wherein each ILC module can be coupled to adjacentrespective processing regions 410-414. Each ILC module 450-456 can alsobe coupled to adjacent respective regions of the first memory 402-408.The inter-layer-communication units 450-456 can be configured tosynchronize data movement between one or more compute cores producinggiven data and one or more other compute cores consuming the given data.Again, the inter-layer communication unit 450-456 can map thecomputation functions of compute cores and dataflow between processingregions 410-414 and first memory 402-408 based on adjacency so thatdataflow of shared data can be synchronized therebetween.

The compute cores of the core groups 434-448 of the processing regions410-414 can include a plurality of physical channels configurable toperform computations, accesses and the like, simultaneously with othercores within respective core groups 434-448 and or processing regions410-414, and or simultaneously with other cores in other core groups434-448 and or processing regions 410-414. The compute cores can alsoinclude a plurality of configurable virtual channels.

Relatively large flat memory regions such as the plurality of firstmemory regions described above with reference to FIGS. 1-3 may not beable to provide enough memory bandwidth to achieve a target performancelevel. Therefore, organizing each of the memory regions 402-408 into aplurality of memory blocks 416-432 and coupling a core group 436 of arespective processing region 410 to a set of memory blocks 416, 418,422, 424 that are proximate to the given core group 436, while notcoupled to memory blocks in the adjacent memory regions that are distalfrom the given core group as described above with reference to FIG. 4 ,can increase memory bandwidth throughput. Accordingly, providing more,but smaller, flat memory bocks by organizing each of the plurality ofmemory regions 402-408 into respective sets of a plurality of memoryblocks 416-432 can provide increased memory bandwidth for increasedperformance. The smaller flat memory blocks can also provide thepotential for better chip layout as compared to larger flat memoryorganizations. However, the increased number of the smaller flat memoryblocks can make adjacency mapping for dataflow more challenging.

Referring now to FIG. 5 , a memory processing unit, in accordance withaspects of the present technology, is shown. The memory processing unit500 can include a first memory 402-408 and a plurality of processingregion 410-414. The first memory can include a plurality of memoryregions 402-408. The plurality of processing regions 410-414 can beinterleaved between the plurality of memory regions 402-408 of the firstmemory. In one implementation, the plurality of first memory regions402-408 and the plurality of processing regions 410-414 can haverespective predetermine sizes. One or more of the plurality of memoryregions 402-408 can include a plurality of memory blocks 416-432. One ormore processing regions 410-414 can also include plurality of coregroups 434-448. A core group 434-448 can include one or more computercores. The computer cores in a respective core group can be arranged inone or more compute clusters. One or more of the plurality of coregroups of a respective one of the plurality of processing regions can becoupled between adjacent memory blocks of adjacent ones of the pluralityof memory regions of the first memory. In one implementation, a givencore group can be coupled to a set of directly adjacent memory blocks,while not coupled to the other memory blocks of the adjacent memoryregions. In other words, a core group of a respective processing regioncan be coupled to a set of memory blocks that are proximate to the givencore group, while not coupled to memory blocks in the adjacent memoryregions that are distal from the given core group. For example, a firstcore group 434 of a first processor region 410 can be coupled between afirst memory block 416 of a first memory region 402 and a first memoryblock 422 of a second memory region 404. A second core group 436 of thefirst processor region 410 can be coupled to the first and a secondmemory block 416, 418 of the first memory region 402 and the first and asecond memory block 422, 424 of the second memory region 404. The secondcore group 436 of the first processor region 410 can also be coupledbetween the first and a third core group 434, 438 of the first processorregion 410.

The memory processing unit 500 can also include a second memory 510. Thesecond memory 510 can be coupled to the plurality of processing regions410-414. The second memory 510 can optionally be logically or physicallyorganized into a plurality of regions (not shown). The plurality ofregions of the second memory 510 can be associated with correspondingones of the plurality of processing region 410-414. In addition, theplurality of regions of the second memory 510 can include a plurality ofblocks organized in one or more macros. The second memory can benon-volatile memory, such as resistive random-access memory (RRAM),magnetic random-access memory (MRAM), flash memory (FLASH) or the like.The second memory can alternatively be volatile memory.

One or more of the compute cores, and or one or more core groups of theplurality of processing regions 410-414 can be configured to perform oneor more computation functions, one or more instances of one or morecomputation functions, one or more segments of one or more computationfunctions, or the like. For example, a first computer core, a first coregroup 434 or a first processing region 410 can be configured to performtwo computation functions, and a second computer core, second core groupor second processing region 412 can be configured to perform a thirdcomputation function. In another example, the first compute core, thefirst core group 434 or the first processing region 410 can beconfigured to perform three instances of a first computation function,and the second compute core, second core group or the second processingregion 412 can be configured to perform a second and third computationfunction. In yet another example, a given computation function can havea size larger than the predetermined size of a compute core, core groupor one or more processing regions. In such case, the given computationfunction can be segmented, and the computation function can beconfigured to be performed on one or more compute cores, one or morecore groups or one or more of the processing regions 410-414. Thecomputation functions can include, but are not limited to, vectorproducts, matrix-dot-products, convolutions, min/max pooling, averaging,scaling, and or the like.

The dataflow can be configured by the one or more centralized ordistributed control circuitry inter-layer communication (ILC) units450-456 to flow between adjacent columnal interleaved processing regions410-414 and memory regions 402-408 in a cross-columnal direction. In oneimplementation, one or more communication links can be coupled betweenthe interleaved plurality of memory region 402-408 and plurality ofprocessing regions 410-414. The one or more communication links can alsobe configured for moving data between non-adjacent ones of the pluralityof memory regions 402-408, between non-adjacent ones of the plurality ofprocessing regions 410-414, or between non-adjacent ones of a givenmemory region and a given processing region.

The plurality of processing regions 410-414 can be configurable formemory-to-core dataflow from respective ones of the plurality of regionsof the first memory 402-408 to one or more cores within adjacent ones ofthe plurality of processing regions 410-414. The plurality of processingregions 410-414 can also be configurable for core-to-memory dataflowfrom one or more cores within ones of the plurality of processingregions 410-414 to adjacent ones of the plurality of regions of thefirst memory 402-408. In one implementation, the dataflow can beconfigured for a given direction from given ones of the plurality ofregions of the first memory 402-408 through respective ones of theplurality of processing regions to adjacent ones of the plurality ofregions of the first memory 402-408.

The plurality of processing regions 410-414 can also be configurable formemory-to-core data flow from the second memory 510 to one or more coresof corresponding ones of the plurality of processing regions 410-414. Ifthe second memory 510 is logically or physically organized in aplurality of regions, respective ones of the plurality of regions of thesecond memory 510 can be configurably couplable to one or more computecores in respective ones of the plurality of processing regions 410-414.

The plurality of processing regions 410-414 can be further configurablefor core-to-core data flow between select adjacent compute cores inrespective ones of the plurality of processing regions 410-414. Forexample, a given core can be configured to pass data accessed from anadjacent portion of the first memory 402 with one or more other coresconfigurably coupled in series with the given compute core. In anotherexample, a given core can be configured to pass data accessed from thesecond memory 510 with one or more other cores configurably coupled inseries with the given compute core. In yet another example, a givencompute core can pass a result, such as a partial sum, computed by thegiven compute core to one or more other cores configurably coupled inseries with the given compute core.

Again, relatively large flat memory regions, such as the plurality offirst memory regions described above with reference to FIGS. 1-3 , maynot be able to provide enough memory bandwidth to achieve a targetperformance level. Therefore, organizing each of the memory regions402-408 into a plurality of memory blocks 416-432 and coupling a coregroup 436 of a respective processing region 410 to a set of memoryblocks 416, 418, 422, 424 that are proximate to the given core group436, while not coupled to memory blocks in the adjacent memory regionsthat are distal from the given core group as described above withreference to FIG. 4 , can increase memory bandwidth throughput.Accordingly, providing more, but smaller, flat memory bocks byorganizing each of the plurality of memory regions 402-408 intorespective sets of a plurality of memory blocks 416-432 can provideincreased memory bandwidth for increased performance. The smaller flatmemory blocks can also provide the potential for better chip layout ascompared to larger flat memory organizations. However, the increasednumber of the smaller flat memory blocks can make adjacency mapping fordataflow more challenging.

The plurality of processing regions 410-414 can include one or more nearmemory (M) compute cores. The one or more near memory (M) compute corescan be configurable to compute neural network functions. For example,the one or more near memory (M) compute cores can be configured tocompute vector-vector products, vector-matrix products, matrix-matrixproducts, and the like, and or partial products thereof.

The plurality of processing regions 410-414 can also include one or morearithmetic (A) compute cores. The one or more arithmetic (A) computecores can be configurable to compute arithmetic operations. For example,the arithmetic (A) compute cores can be configured to compute mergeoperations, arithmetic calculations that are not supported by the nearmemory (M) compute cores, and or the like.

A plurality of input and output regions (not shown) can also include oneor more input/output (I/O) cores. The one or more input/output (I/O)cores can be configured to access input and or output ports of thememory processing unit (MPU) 500. The term input/output (I/O) core asused herein can refer to cores configured to access input ports, coresconfigured to access output ports, or cores configured to access bothinput and output ports.

The compute cores can also include other types of compute cores such asgraph processing cores or the like. The compute cores of the core groups434-448 of the processing regions 410-414 can include a plurality ofphysical channels configurable to perform computations, accesses and thelike, simultaneously with other cores within respective core groups434-448 and or processing regions 410-414, and or simultaneously withother cores in other core groups 434-448 and or processing regions410-414. The compute cores can also include a plurality of configurablevirtual channels.

The plurality of memory regions 402-408 can also be organized into aplurality of memory blocks arranged in a plurality of columns and rowsfor each memory region 402-408. For example, each given memory region404 can be organized into a plurality of memory blocks of m blocks wideand n blocks long, wherein m and n can be different or equal. A fetchunit for a respective processing region, core group or compute core canbe configured to fetch from sets of memory blocks of respective adjacentmemory regions. Similarly, a write back unit for a respective processingregion, core group or compute core can be configured to write back to aset of memory blocks of respective adjacent memory regions. Theorganization of the plurality of memory blocks in a plurality of columnsand rows can provide further increased memory bandwidth for increasedperformance. The organization of the plurality of memory blocks arrangedin a plurality of columns and rows is further explained below withreference to FIGS. 53A-53D and 54A-54C.

In accordance with aspects of the present technology, a neural networklayer, a part of a neural network layer, or a plurality of fused neuralnetwork layers can be mapped to a single cluster of compute cores or acore group as a mapping unit. A cluster of compute cores is a set ofcores of a given processing region that are configured to work togetherto compute a mapping unit. For example, the nodes of a first layer 610of a neural network can be mapped as a mapping unit to a first set ofcompute cores, the nodes of a second layer 620 can be mapped to a secondset of compute cores, while the node of a third layer 630 can be mappedto a third set of compute cores, as illustrated in FIG. 6 . Furthermore,a mapping unit 710 can be computed by a compute core cluster 720 asillustrated in FIG. 7 . Optionally, more compute cores than are neededto compute a mapping unit can be configured in a compute cluster toimprove computing performance.

Referring now to FIGS. 8A-8B, an exemplary computation of multipleoutput feature map pixels, in accordance with aspects of the presenttechnology, is illustrated. One or more compute cores can be configuredto compute a corresponding output feature map pixel from an inputfeature map pixel value and a kernel data (weight) value. Asillustrated, compute cores can be configured as three pixel workers tocompute output feature map pixel values for each of the output channels.For example, a given pixel worker can compute output feature map pixelvalues 810-850 for each of the output channels of the output featuremap. The pixel workers can then step to the next set of three pixelvalues to compute the corresponding output channels of the outputfeature map, as illustrated in FIG. 8B. In a polymorphic implementation,multiple compute cores can work together as pixel workers. The maximumnumber of pixel workers for a given layer is limited to the outputfeature map width of the given layer. The kernel, weight data or thelike can be reused without reloading them from the second memory region.

Referring now to FIG. 9 , configuration of dataflows in the memoryprocessing unit, in accordance with aspects of the present technology,is illustrated. The dataflow 910, 920 between the compute cores 950-956of the processing regions and adjacent regions of first memory 960, 962can be configured in either direction. For example, the compute cores950-956 and the adjacent portions of the first memory region 960, 962can be configured for dataflow from a first region of the first memory960, through the compute cores 950-956, and to a second region of thefirst memory 962. Alternatively, the dataflow can be configured from thesecond region of the first memory 962, through the compute cores950-956, to the first region of the first memory 960. In oneimplementation, the dataflow between the compute cores 950-956 of theprocessing regions and adjacent regions of first memory 960-962 canprovide a direct route to access feature map data or the like.

The dataflow 930 from the second memory 970 to the compute cores of theprocessing regions can also be configured. In one implementation, thedataflow from the second memory 970 to the compute cores 950-956 canprovide a direct route to access kernel data, weight data or the like.The dataflow 940 between the compute cores 950-956 can also beconfigured. In one implementation, the dataflow between the computecores 950-956 can provide for the sharing of data from the second memorywith others of the compute cores 950-956 in a corresponding core groupand or processing region.

The plurality of processing regions can include one or more near memory(M) compute cores, one or more arithmetic (A) compute cores, and one ormore input/output (I/O) cores. The one or more near memory (M) computecores can be configurable to compute neural network functions. The oneor more arithmetic (A) compute cores can be configurable to computearithmetic operations. The one or more input/output (I/O) cores can beconfigured to access input and or output ports of the memory processingunit (MPU).

Referring now to FIG. 10 , a near memory (M) compute core, in accordancewith aspects of the present technology, is shown. The near memory (M)compute core 1000 can include a fetch unit 1005, amultiply-and-accumulate (MAC) array unit 1010, a writeback unit 1015 anda switch 1020. The fetch unit 1005 can be configured to fetch data froman N^(th) portion of the first memory for the multiply-and-accumulate(MAC) array unit 1010. The fetch unit 1005 can also be configured toreceive data from a N−1^(th) compute core and or pass data to a N+1^(th)compute core within a respect processing region. The fetch unit 1005 canalso be configured to receive data from the second memory. The fetchunit 1005 can also be configured to synchronize data movement the N^(th)portion of the first memory with the inter-layer-communication (ILC)unit. In one implementation, the fetch unit 1005 can be configured tocontrol an operation sequence of the near memory (M) compute core 1000,to fetch data from the second memory or an adjacent one of a sequence ofthe plurality of compute cores in a respective processing region, tofetch data from an adjacent one of the plurality of regions of the firstmemory, to decrement an inter-layer-communication (ILC) counter, and totrigger other units of the near memory (M) core.

The multiply-and-accumulate (MAC) array unit 1010 can be configured tocompute neural network functions. For example, themultiply-and-accumulate (MAC) array unit 1010 can be configured tocompute vector-vector products, vector-matrix products, matrix-matrixproducts, and the like, and or partial products thereof. Themultiply-and-accumulate (MAC) array unit 1010 can also be configured toperform pre-channel and bias scaling. In one implementation, themultiply-and-accumulate (MAC) array unit 1010 can be configured toperform main operations such as, but not limited to, dense or fullyconnected convolutions, two-dimensional convolutions, depth-wiseconvolutions, and separable convolutions. The multiply-and-accumulate(MAC) array unit 1010 can also be configured to perform fused operationssuch as, but not limited to, max pooling, average pooling, rectifylinear (ReLU) activation, ReLU-x activation, and up-sampling. Themultiply-and-accumulate (MAC) array unit 1010 can also be configured toperform virtually fused operations such as, but not limited to, zeropadding (folded into kernel corners), average pooling (folded intoweights and biases), ReLU activation, ReLU-x activation, andup-sampling.

The writeback unit 1015 can be configured to write data to an N+1^(th)portion of the first memory for the multiply-and-accumulate (MAC) arrayunit 1010. The writeback unit 1015 can also be configured to synchronizedata movement the N^(th) portion of the first memory with theinter-layer-communication (ILC) unit. In one implementation, thewriteback unit 1015 can be configured to perform a fuse operation, senddata to an adjacent region of the first memory or adjacent compute corein the respective processing region, and to increment aninter-layer-communication (ILC) counter.

The switch 1020 can configure memory accesses, and chain directions andinterfaces of the fetch unit and writeback units to ports of therespective near memory (M) compute core based on configurationinformation. The switch 1020 can be preconfigured with memory access andchain directions. The switch 1020 can therefore interface the fetch 1005and writeback units 1015 based on the data-flow configuration.

The near memory (M) compute core 1000 can include a plurality ofphysical channels configurable to perform computations simultaneously.The near memory (M) compute core 1000 can also be associated with one ormore blocks of the second memory. The physical channels of the nearmemory (M) compute core 1000 can be associated with respective slices ofthe second plurality of memory regions. The near memory (M) compute core1000 can also include a plurality of configurable virtual channels.

Referring now to FIG. 11 , an arithmetic (A) compute core, in accordancewith aspects of the present technology, is shown. The arithmetic (A)compute core 1100 can include a fetch unit 1105, an arithmetic unit1110, a writeback unit 1115 and a switch 1120. Again, the fetch unit1105 can be configured to fetch data from an N^(th) portion of the firstmemory for the arithmetic unit 1110. The fetch unit 1105 can also beconfigured to synchronize data movement the N^(th) portion of the firstmemory with the inter-layer-communication (ILC) unit. In oneimplementation, the fetch unit 1105 can be configured to control anoperation sequence of the arithmetic unit 1110, to fetch data from anadjacent one of the plurality of regions of the first memory, decrementan inter-layer-communication (ILC) counter, and trigger other units ofthe arithmetic (A) compute core 1100.

The arithmetic unit 1110 can be configured to compute arithmeticoperations not supported by the multiply accumulate (MAC) array unit1010. For example, the arithmetic unit 1110 can be configured to computemerge operations and or the like. The arithmetic unit 1110 can computeone or more output channels at a time. The arithmetic unit 1110 may nothave access to the second memory. The arithmetic unit 1110 may have nomeans to pass data between adjacent cores in the same processing region.In one implementation, the arithmetic unit 1110 can be configured toperform main operations such as, but not limited to, add, multiply andbypass. The arithmetic unit 1110 can also be configured to fuseoperations such as, but not limited to, ReLU activation, ReLU-xactivation, and leaky ReLU-x activation.

The writeback unit 1115 can be configured to write data to an N+1^(th)portion of the first memory for the arithmetic unit 1110. The writebackunit 1115 can also be configured to synchronize data movement the N^(th)portion of the first memory with the inter-layer-communication (ILC)unit. In one implementation, the writeback unit 1115 can be configuredto perform a fuse operation, send data to an adjacent region of thefirst memory or an adjacent compute core in the respective processingregion, and to increment an inter-layer-communication (ILC) counter.

The switch 1120 can be configured to configure memory accesses, chaindirections and interfaces of the fetch unit and writeback units to portsof the arithmetic compute core based on configuration information.

Referring now to FIG. 12 , an input (I) core, in accordance with aspectsof the present technology, is shown. The input (I) core 1200 can includean input port 1205, a writeback unit 1210 and switch 1215. The inputport 1205 can be configured to receive data into the memory processingunit and trigger the writeback unit 1210. The writeback unit 1210 can beconfigured to stream the received data into a first portion of the firstmemory and increment an inter-layer-communication (ILC) counter. Theswitch 1215 can be configured to connect the writeback unit 1210 to theadjacent regions of the first memory based on configuration information.In one implementation, an input stage can be comprised of a single ormultiple input (I) cores 1200.

Referring now to FIG. 13 , an output (O) core, in accordance withaspects of the present technology, is shown. The output (O) core 1300can include a fetch port 1305, an output unit 1310 and a switch 1315.The fetch port 1305 can be configured to stream data out from a lastportion of the first memory and trigger the output unit 1310. The outputunit 1310 can be configured to output data out of the memory processingunit. The switch 1315 can be configured to connect the fetch port 1305to the adjacent regions of the first memory and theinter-layer-communication (ILC) unit based on configuration information.In one implementation, an output stage can be comprised of a single ormultiple output (O) cores 1300.

Referring now to FIG. 14 , a whole channel compute core configuration,in accordance with aspects of the present technology, is shown. Thecompute cores, of a given processing region, can be configured in wholechannel mode, wherein one or more compute cores perform computationsindependently of the other compute cores in a respective processingregion. In the whole channel mode, the compute cores do not pass data1410 sequentially from a given compute core to an adjacent compute core.Referring now to FIG. 15 , in the whole channel mode, each compute corein the cluster computes a designated number of channels. Each of thecores is responsible for reading data and writing the output result ontheir own. For example, a whole channel mode configured compute corereads data from the X^(th) portion of the first memory region, andoptionally the second memory region, performs a correspondingcalculation and stores the result in the (X+1)^(th) portion of the firstmemory region. The compute cores in whole channel mode do not share datawith other compute cores and work as standalone compute cores. Referringnow to FIG. 16 , an exemplary whole channel compute core configurationis illustrated. In the illustrated example, the mapping unit has 22output channels 1610 and is mapped to a three-compute core cluster1620-1640. Each compute core has four output physical channels. An inputfeature map 1650 is stored in an adjacent first portion of the firstmemory region, and an output feature map 1660 is stored in an adjacentsecond portion of the first memory region. As further illustrated inFIG. 17 , each compute core 1620-1640 is configured to access weightsfor the respective output channels. Each compute core is configured tocompute a product of the input feature map and the weights of respectivesets of the 22 output channels 1710 of the output feature map. Eachcompute core is responsible for almost one-third of the computationworkload. The second memory region can be organized based on outputchannels, and result in the 22 output channels 1710 mapped into five andhalve virtual channel rows. Although, the compute core cluster isillustrated as mapped over a single macro of the second memory region,the compute core cluster can also be mapped over a plurality of macrosof the second memory region.

Referring now to FIG. 18 , a polymorphic second memory compute coreconfiguration, in accordance with aspects of the present technology, isshown. The compute cores, of a given processing region, can beconfigured in a polymorphic configuration, wherein one or more computecores share data from a given portion of the second memory region 1810with adjacent compute cores. In the polymorphic second memory computecore configuration, each compute core of the cluster can compute all theoutput channels, but work on different pixels of an output feature map.Accordingly, the other compute cores in the cluster operate as workersfor the first compute core. The number of compute cores that can beassigned is the number of mapping unit output feature map pixels. Thecompute cores of the cluster access a different sequence of data in thesecond memory region since they are working on different pixels. Such aconfiguration can be used to reduce the number of access to the secondmemory region by sharing the data among cores in the cluster. The firstcompute core 1910 in a polymorphic second memory cluster has access todata in the corresponding portion of the second memory region 1940 andcan share the data with the other compute cores 1920, 1930 in thecluster. All the compute cores 1910-1930 in the polymorphic secondmemory cluster have access to data in the first memory region 1950, andall of the compute cores 1910-1930 can write results to the otheradjacent portion of the first memory region 1960, as illustrated in FIG.19 . Referring now to FIGS. 20 and 21 , an exemplary polymorphic secondmemory compute core configuration is illustrated. In the illustratedexample, the compute cores 2010-2030 of a cluster can all access inputfeature map data in a first adjacent portion of the first memory region2040, as illustrated in FIG. 20 . The first compute core 2010 can accessdata in the second memory region 2110, and share the data with the othercompute cores of the cluster 2020, 2030, as illustrated in FIG. 21 . Inone implementation, the cluster can include 3 compute cores 2010-2030mapped with a total of 22 output channels. Each compute core can havefour physical channels 2120. The top compute core 2010 of the chain isassigned the whole portion of the second memory region 2110 needed bythe mapping, and access the whole 22 output channels of data. Eachcompute core computes all 22 output channels, but for different pixels.The other two compute cores 2020, 2030 of the cluster will access thefirst compute core 2010 rather than the second memory region 2110 to getweight data. The neighbor access can be done in a dataflow mannerwithout special synchronization. Each compute core 2010-2030 in thecluster can then perform a respective computation and write the resultsas output feature map data to the other adjacent portion of the firstmemory region 2050, as illustrated in FIG. 20 .

Referring now to FIG. 22 , a polymorphic first memory compute coreconfiguration, in accordance with aspects of the present technology, isshown. The compute cores, of a given processing region, can beconfigured in a polymorphic configuration, wherein one or more coresshare data from a given portion of the first memory region 2210 withadjacent compute cores. The polymorphic first memory compute coreconfigured cluster is equivalent to a wider core with more physicalchannels. Such a configuration can be used to improve reuse of data inthe first memory region and reduce the total number of accesses to thecorresponding portion of the first memory region. It should also benoted that reuse of data in the first memory region is also an inherentproperty of the compute core configuration of the plurality ofprocessing region in accordance with aspects of the present technologybecause the compute cores can share data among the physical channels.The first compute core 2310 in a polymorphic first memory computecluster has access to data in the corresponding portion of the firstmemory region 2340 and can share the data with the other compute cores2320, 2330 in the cluster. All the compute cores 2310-2330 in thepolymorphic first memory configuration have access to data in the secondmemory region 2350, and all of the compute cores 2310-2330 can writeresults to the other adjacent portion of the first memory region 2360,as illustrated in FIG. 23 . Referring now to FIGS. 24 and 25 , anexemplary polymorphic first memory region compute core configuration isillustrated. In the illustrated example, the first compute core 2410 ofa cluster can access input feature map data in a first adjacent portionof the first memory region 2440. The first compute core 2410 can sharein the data of the input feature map with the other compute cores 2420,2430 of the cluster, as illustrated in FIG. 24 . Each compute core2410-2430 in the cluster can also access data in the second memoryregion 2510, as illustrated in FIG. 25 . Each compute core 2410-2430 inthe cluster can then perform a respective computation and write theresults as output feature map data to the other adjacent portion of thefirst memory region 2450, as illustrated in FIG. 24 . The polymorphicfirst memory compute cluster can be configured by a mapping algorithmthat starts by creating a whole-channel cluster, then converting to thefirst memory region polymorphic computer cluster. In the illustratedthree compute core cluster, each core can be responsible for up to onethird of the computer workload. The second memory region 2510 can beconfigured to have four output channels, that can be mapped into fiveand a half virtual channel rows in the second memory region 2510, asillustrated in FIG. 25 .

Referring now to FIG. 26 , a compound compute core configuration, inaccordance with aspects of the present technology, is shown. Eachcompute core in a cluster, of a given processing region, can access anadjacent portion of the first memory region. The compute cores can alsobe configured to share data from a given portion of the second memoryregion 2610, 2620 with adjacent compute cores within the same set.However, compute cores in different sets do not share 2630 data withother compute cores in other sets. The compute cores ©n each set computea designated number of output channels and store results into the otheradjacent portion of the first memory region. Referring now to FIG. 27 ,an exemplary compound compute core configuration is illustrated. In theillustrated example, the mapping unit has 22 output channels and ismapped to a four-compute core cluster 2710-2840 including two sets oftwo compute cores each. For example, a first set can include first andsecond compute cores 2710, 2720, and a second set can include third andfourth compute cores 2730, 2740. Each set of compute cores can have fourphysical channels per core. Each compute core 2710-2740 in each set canaccess input feature map data in the first memory 2750 as illustrated inFIG. 28 . The first compute cores 2710, 2730 in a respective set canaccess weight data in a respective set of output channels in the secondmemory 2770, as illustrated in FIG. 29 . The first compute core in afirst set 2710 can be configured to share data from the second memory2770 with the other compute cores in the first set 2720. Similarly, afirst compute core in a second set 2730 can be configured to share datafrom the second memory 2770 with the other compute cores in the secondset 2740. Each compute core 2710-2740 of each set can store result backas output feature map data to the other adjacent portion of the firstmemory 2760. Accordingly, each set of two compute cores act asstand-alone pixel computing groups. However, the whole result iscomputed using the two sets of pixel computing groups. At a top level,each of the pixel computing groups can be treated as a standalonecompute core set, and the workload can be distributed between them in awhole-channel way.

Referring now to FIG. 30 , a first memory region sharing feature of thememory processing unit (MPU), in accordance with aspects of the presenttechnology, is shown. As illustrated, the dataflow of computations bythe MPU can be visualized as a series of produces 3010-3040 andconsumers 3050-3070. For example, a compute core cluster 3010-3040 canconsume input feature map data from a first portion of the first memoryregion and produce feature map data that can be an input to a nextcompute core cluster 3050-3070 to use. It is to be appreciated that datasharing in general between conventional computing units tends to be asignificant obstacle to conventional dataflow accelerators. Therefore,conventional processing units may utilize network-on-chip and or dataduplications. In contrast, the MPU in accordance with aspects of thepresent technology enables a much simpler data sharing technique,wherein producers and consumers write and read to a shared memory buffer3080. The buffers 3080 are interleaved portions of the first memorybetween the plurality of processing regions. Accordingly, data can flowbetween clusters in the same processing region and or adjacentprocessing regions. In one implementation, a software layer can beconfigured to organize the clusters to ensure such adjacency. In theexample of FIG. 30 , two compute core clusters 3010-3040 and 3050-3070in two different processing regions share a buffer 3080 in a portion ofthe first processing region. It is to be appreciated that there is nodirect communication between the producer and the consumer computecores. Compute cores in a compute cluster do not directly synchronizewith each other. However, compute cores in a compute cluster can beconfigured to directly communicate data with each other.

In one implementation, data can be shared between processing regions byassigning a large enough buffer in the corresponding portion of thefirst memory. For example, the buffer can be allocated to carry a wholefeature map shared between adjacent processing regions. The size of thebuffer can be calculated in accordance with Equation 1:

S _(b)=Π_(∀i) F[i]  (1)

where F is the vector of the feature map size.

However, assigning the whole feature map size as a buffer is not enoughfor the data to flow. Consumers need to avoid reading a buffer entrythat is not filled yet by the producer. Assuming a coarse-grainsynchronization of the feature map row level, the consumer cannot readfrom a feature map row that is still being produced. For the sake ofsimplicity, each feature map row will be illustrated as a single bufferentry in FIGS. 31-36 . However, it is appreciated that a single row mayrequire the storage of hundreds, thousands, or even more entries.Referring now to FIGS. 31A and 31B, an exemplary buffer utilization by aconsumer and a producer is illustrated. The illustrated buffer 3110 issized to store a full feature map. The producer 3120, for example, canbe performing a two-dimensional convolution, and the consumer 3130 canbe performing a two-dimensional convolution having a 3×3 kernel size.The producer core 3120 can generate the pixels of a given feature maprow before producing the pixels of a next row. In such case, theproducer core 3120 only blocks a single row entry as illustrated in FIG.31A. As the producer core 3120 generates the pixels of a given featuremap row, the consumer core 3130 can access the pixels values of theprevious three rows. After the producer core 3120 is done generating thepixels of the given row, the producer core 3120 can move to generate thepixels of the next row as illustrated in FIG. 31B. At that point, theconsumer core 3130 can shift its consumption to a next three row windowif the consumer core 3130 is ready to start processing the next threerow window. Furthermore, it is noted that the rows that have alreadybeen consumed can remain in the buffer 3110 until overwritten by theproducer core 3120 as processing continues. It is appreciated that theconsumer 3130 of a 3×3 kernel consumes three buffer entriessimultaneously while the producer 3120 generates data for one entrybefore moving to the next one. Furthermore, a number of entries in thebuffer 3110 are not in use at any given time. Therefore, the fullfeature map sized buffer 3110 can waste resources in the memoryprocessing unit (MPU).

In another implementation, a smaller partial buffer can be sufficientfor the dataflow to support the computations. For example, a circularqueue can be utilized as a partial buffer. The partial buffer can beconfigured to carry enough data for the consumer to operate and haveextra entries to allow the producer to generate data while the consumeris working. For example, the partial buffer can include three featuremap rows in the case where the consumer is performing a convolutionhaving a 3×3 kernel size. The partial buffer can also include extraentries, referred to as a pipeline margin. Without such a margin, thedataflow performance will degrade significantly since the producer andconsumer will not be able to work concurrently. The producer also cannotoverwrite data that is not yet consumed, and the consumer needs to waitfor the producer to finish writing a new row in the partial bufferbefore starting to consume it. Referring now to FIGS. 32A-32D, anexemplary shared partial buffer 3410 for a 3×3 kernel size isillustrated. As illustrated, a producer 3220 generates pixel data for agiven row before moving on to the next row, and the consumer 3230accesses three rows of data at a time. By utilizing a partial buffer3210, the size of the shared buffer 3210 can be reduced to as littles asfour rows. For example, in a first cycle the consumer 3230 can beaccessing the first three rows of pixel data, and the producer 3220 canbe generating data for storing in the fourth row. In a second cycle, theconsumer 3230 can be accessing the second through four rows of data,while the producer 3220 is storing data in the first row. In a thirdcycle, the consumer 3230 can access data in the third, fourth and firstrows, while the producer 3220 stores data in the second row. In a fourthcycle, the consumer 3230 can access the fourth, first and second rows,while the producer 3220 stores data in the third row. Thereafter, thefirst through fourth cycles can be iteratively repeated any number oftimes. Accordingly, the four-row shared partial buffer can allow theproducer and consumer to work smoothly.

Referring now to FIGS. 33A and 33B, an exemplary shared partial bufferfor a 3×3 kernel size with a 2×2 stride is illustrated. A consumer 3330having a stride of 2×2 moves its window two rows at a time. Therefore, apipeline margin of two is needed to allow the producer to generate thenecessary rows for the consumer window shift. For example, a producer3320 can store data in a fourth and fifth row, while the consumer 3330accesses data in the first through third rows. After the producer 3320stores data in the fourth and fifth rows, the consumer 3330 can move toaccessing data in the third through fifth rows, while the producer 3320stores data in the first and second rows.

For ease of explanation, aspects of the present technology have beendescribed with regard to a single producing cluster and a singleconsuming cluster. However, dataflow in the memory processing unit (MPU)can involve dataflow branching into multiple paths that can for exampleend as different outputs, merge again, and the like. While branchingoutput can be treated the same as multiple single dataflow paths,merging branches can involve additional considerations. If a neuralnetwork with merging branches, for example, is not allocated the correctbuffer size, the dataflow pipeline might end up in a deadlock or produceincorrect data. With data having multiple consumers, the data validityshould be set by the slowest consumer. Typically, a longer data lifetimeresults in a need for a larger buffer size. Referring now to FIG. 34 ,an example branching dataflow utilizing a full feature-map buffer isillustrated. As illustrated, a first producer 3410 can perform aconvolution (Conv2D) operation, which is consumed by two branches. Afirst branch, can for example, include a series of two convolution(Conv2D) operations 3420, 3430 of a kernel size of 3×3. A second branchcan include a skip connection 3440, for example. The two branches canthen be merged together, for example, with the aid of an addition (Add)operation 3450. Each of the convolution (Conv2D) operations 3420, 3430in the first branch need three ready rows to access for input data. Inaddition, an extra row is also needed as a pipeline margin. The addoperation 3450 does not have any kernels and therefore only needs asingle ready row to operate. However, the producer data cannot beoutdated based on the convolution (Conv2D) consumers 3420, 3430. Dataneeds to stay in the buffer until the Add merge node 3450 is ready touse it.

Referring now to FIG. 35 , an exemplary branching dataflow utilizing apartial feature-map buffer is illustrated. As illustrated, the producer3510 at the start of the branch produces two sets of data for consumers(with the aid of bypass operations) of the two branches to facilitatedata synchronization. The faster branch is configured to buffer 3520more data to align with the slower branch, which can be referred to asthe branch delay data. It is to be appreciate that not all branchesrequire a delay buffer. For example, balanced branches do not requireextra data storage, as illustrated in FIG. 36 . As illustrated, each ofthe two branches can be configured with a typical size of partial bufferas if each branch is the only data path.

The inter-layer-communication (ILC) unit can be configured tosynchronize data movement between one or more compute cores producinggiven data and one or more other compute cores consuming the given data.Data communication within the memory processing unit can include directand indirect connections between two modules. Direct synchronization canbe implemented by direct wire connections with a producer/consumerhandshake. The direct synchronization can be implemented by polymorphicconnections between compute cores.

The inter-layer-communication unit can also synchronize indirectconnections between two modules. Indirect synchronization can beimplemented by use of a buffer between two modules. Indirectsynchronization by the inter-layer-communication unit can be implementedas communication between compute cores and volatile memory (e.g., SRAM).In such an implementation, a producer compute core can write to a sharedbuffer in a corresponding first memory region and a consumer computecore can read from the shared buffer. The data can be synchronized toavoid data hazards that can occur in the buffer. Exemplary data hazardscan include a producer core overwriting data to a buffer before aconsumer core can read data from the buffer, or a consumer core readingdata from a buffer before the producer core can write the data to thebuffer. In one implementation, indirect synchronization can beimplemented by the compute cores sending appropriate signals to thebuffer to provide visible synchronization. In visible indirectsynchronization, the buffers between the compute cores can act as asimple memory used for writing and reading data. The producer core canbe configured to ensure that the consumer core is ready for data, andthe consumer core can be configured to ensure that there is enough datain the memory so that it can perform a computation operation.

In another implementation, indirect synchronization can be implementedby the ILC unit to provide invisible synchronization. In the invisibleindirect synchronization the ILC unit is responsible for keepingproducer compute cores and consumer compute cores in synchronization.

Referring now to FIG. 37 , a memory processing unit (MPU), in accordancewith aspects of the present technology, is shown. The memory processingunit can include a first memory including a plurality of regions3705-3710, a plurality of compute cores 3715-3755 organized in aplurality of processing regions, a second memory (not shown) and aninter-layer-communication (ILC) unit 3760-3765. The memory processingunit MPU can be arranged as described above with reference to FIGS. 2-5. In one implementation, the layer-communication (ILC) unit 3760-3765can include a plurality of layer-communication (ILC) modules, whereineach layer-communication (ILC) module 3760, 3765 controls data movementthrough a corresponding regions of the first memory 3705, 3710, betweenone or more compute cores producing given data and one or more othercompute cores consuming the given data.

In one implementation, data flow between compute cores 3715-3725 of oneor more of a plurality of processing regions and corresponding adjacentones of the plurality of regions of the first memory 3705 can beconfigured utilizing direct synchronization between the compute coresand the first memory. Similarly, data flow between the second memory(not shown) and the compute cores 3715-3755 of the one or more of theplurality of processing regions can be configured utilizing directsynchronization between the compute cores 3715-3755 and the secondmemory. Data flow between compute cores 3715-3725 within respective onesof the one or more of the plurality of processing regions can also beconfigured utilizing direct synchronization between adjacent computecores within the respective processing region.

The inter-layer-communication (ILC) unit 3760-3765 can synchronize datamovement between one or more compute cores 3715-3725 producing givendata and one or more other compute cores 3730-3740 consuming the givendata utilizing indirect invisible synchronization. Data movementsynchronization by the inter-layer-communication (ILC) unit 3760-3765will be further described with reference to FIGS. 38-40 . Referring nowto FIG. 38 , an inter-layer-communication method, in accordance withaspect of the present technology, is shown. Theinter-layer-communication (ILC) unit 3760-3765 can be configured toreceive synchronization commands related to respective buffers 3770 ofrespective ones of the plurality of regions of the first memory 305 fromrespective compute cores 3715-3755 of the plurality of processingregions, at 3810. For example, inter-layer-communication (ILC) unit3760-3765 can receive synchronization commands from a first one 3720 ofthe plurality of compute cores 3715-3755 related to writing data to ashared buffer 3770 in a first portion of the first memory 3705. In oneimplementation, a producer compute core can send an incrementsynchronization command when it finishes writing a whole feature-memoryrow to the buffer. The inter-layer-communication (ILC) unit 3760-3765can also receive access commands from a second one 3730 of the pluralityof compute cores 3715-3755 related to reading data from the sharedbuffer 3770 in a first portion of the first memory 3705. In oneimplementation, a consumer compute core can send a decrementsynchronization command when it finishes reading a whole feature-memoryrow from the buffer.

At 3820, the inter-layer-communication (ILC) unit 3760-3765 can trackread and write accesses to the respective buffers of respective ones ofthe plurality of regions of the first memory. In one implementationtracking is done on a coarse grain level, such as a whole feature-maprow level. In one implementation, the inter-layer-communication (ILC)unit 3760-3765 can track access to respective buffers with correspondingrespective indexes to point to an ILC entry. Theinter-layer-communication (ILC) unit 3760-3765 does not need to storebuffer region boundaries or other information about the buffer. Instead,the compute cores 3715-3755 can be responsible for accessing the correctILC entry index that corresponds to a respective shared buffer. In oneimplementation, an identifier of a given compute core 3720 received inan synchronization command can be mapped to a count associated with agiven region (e.g., buffer) of a given portion of the first memory 3705.

Referring now to FIG. 39 , respective shared buffers 3910-3930 andcorresponding respective ILC entry indexes 3940-3960, in accordance withaspects of the present technology, are shown. Each ILC entry index caninclude a count of the number of synchronization units that one or moreproducer compute cores have produced (e.g., written) to thecorresponding respective shared buffer, and one or more consumer computecores have yet to consume (e.g., read) from the corresponding respectiveshared buffer. In one implementation, the MC entry index can include acurrent unit count (i_(c)), a maximum count (i_(x)), a minimum count(i_(y)), and an initial count (i_(o)).

At 3830, the inter-layer-communication (ILC) unit 3760-3765 can controlaccess to the buffers of the respective one of the plurality of regionsof the first memory 3750, 3710 by the respective compute cores 3715-3755based on the respective read and write accessing tracking associatedwith the buffers. In one implementation, the inter-layer-communication(ILC) unit 3760-3765 can allow or block requests to a correspondingrespective shared buffer 3770 from one or more respective producercompute cores 3720 and one or more respective consumer compute cores3740 based on the corresponding ILC entry index. For example, theinter-layer-communication (ILC) unit 3760-3765 can allow write access toa respective shared buffer 3770 as long as the current unit count(i_(c)) in the corresponding MC entry index is less than the maximumcount (i_(x)). If the given write access is allowed, theinter-layer-communication (ILC) unit 3760-3765 increments the currentunit count (i_(c)) by an amount of units (i₊) for the given writeaccess, as illustrated in FIG. 40 . If the current unit count (i_(c)) inthe corresponding MC entry index is greater than or equal to the maximumcount (i_(x)), the inter-layer-communication (ILC) unit 3760-3765 blocksthe given write access to the respective shared buffer 3770, and doesnot increment the current unit count (i_(c)). Similarly, theinter-layer-communication (ILC) unit 3760-3765 can allow read access toa respective shared buffer 3770 as long as the current unit count(i_(c)) in the corresponding ILC entry index is greater than the minimumcount (i_(n)). If the given read access is allowed, theinter-layer-communication (ILC) unit 3760-3765 decrements the currentunit count (i_(c)) by an amount of units (i⁻) for the given read access.If the current unit count (i_(c)) in the corresponding ILC entry indexis less than or equal to the minimum count (i_(n)), theinter-layer-communication (ILC) unit 3760-3765 blocks the given readaccess to the respective shared buffer 3770, and does not decrement thecurrent unit count (i_(c)). The difference between the initial count(i_(o)) and the minimum count (i_(n)) represents the amount of data thatmust be produced (written to the corresponding shared buffer) by one ormore producer compute cores before one or more consumer compute coresmay start to consume data from the corresponding shared buffer. If thereare multiple producer compute cores writing to the same shared buffer,the inter-layer-communication (ILC) unit 3760-3765 may require multipleincrement synchronization commands for the compute cores beforeincrementing the current unit count (i_(c)). Furthermore, theinter-layer-communication (ILC) unit 3760-3765 may need to know from thecorresponding computer core when a new data set, such as a new featuremap, is received to reset the counter values. Similarly, as computecores reach the end of a data set, such as a feature map, as indicatedby the current unit count (i_(c)) reaching a “o” value, theinter-layer-communication (ILC) unit 3760-3765 can consider the nextwrite command to be the start of a new data set, such as a feature mapframe.

Referring now to FIG. 41 , a 4-dimension array, in accordance withaspects of the present technology, is illustrated. In oneimplementation, the 4-dimension array may be a weight array utilized inartificial intelligence computations, such as but not limited toconvolution neural network computations. In one implementation, the4-dimensional array can be utilized in 2-dimension convolution layers ofa neural network model. The 4-dimension array can be characterized by akernel width (S), a kernel height ©, input channels © and outputchannels (M) (e.g., number of kernels per layer). Accordingly, thefilters (or kernels) have a dimension of R×S×C, and there are M filters.

Referring now to FIG. 42 , a 3-dimension array, in accordance withaspects of the present technology, is illustrated. In oneimplementation, the 3-dimension array can be utilized in a 2-dimensionaldepth-wise convolution layer of a neural network model. The3-dimensional array can be characterized by a kernel width (S), a kernelheight © and input channels ©. Each kernel has a dimension of R×S, andacts on each input channel separately to produce an output feature mapwith C output channels.

Referring now to FIG. 43 , a 2-dimension array, in accordance withaspects of the present technology, is shown. In one implementation, the2-dimension array can be a dense weight array utilized in a fullconnected layer of a neural network model. The 2-dimension array can becharacterized by flattened input channels © and output channels (M). The2-dimension weight array is typically used in the end of a neuralnetwork mode for classification layers.

Referring to FIG. 44 , a memory macro of a memory processing unit (MPU),in accordance with aspects of the present technology, is shown. Thememory macro appears as a large 2-dimensional memory array. The memorymacro can be characterized by a height and a width. The width of thememory macro can be configured to provide a very wide word fetch. Thewidth of the memory macro can be many words per read wide, which can bedetermined by a needed read bandwidth access for weight arrays. In anexemplary implementation, the access bandwidth of a memory macro can beup to 1024 bits. The height of the memory macro can be a 1-dimensionaladdressable space. The height of the memory macro can be determined bythe total size of the memory macro divided by the width of the memorymacro. The memory macro can be logically split into a plurality ofphysical channels 4410. Each physical channel can be considered a“weight prefetch” wide 4420.

Storage of weight arrays in the memory macros, in accordance withaspects of the present technology, can be configured to improve theperformance of the memory processing unit (MPU). One or more memorymacros can be configured to store all the weights needed for access bythe compute cores of a given group. The one or more memory macros can beconfigured to provide enough memory access bandwidth for the computecores in a given group. The memory macros can be optimized for readaccess by the compute cores. The number of internal memory banks,arrangement and the like of the memory can be transparent to thearchitectural design of the memory processing unit (MPU).

Referring again to FIGS. 41-43 , the weight arrays can be organized forstorage in memory macros to improve performance of a memory processingunit (MPU). The arrangement of weight arrays can impact data throughput,memory utilization, data reuse, memory access pattern, and mapping.Aspects of the present technology can fit a 4-dimension weight arrayinto a 2-dimension memory macro. Aspects of the present technology canalso expand 3-dimension and 2-dimension arrays to look like 4-dimensionarrays for storage in 2-dimension memory macros.

Referring now to FIG. 45 , a method of fitting arrays into a 2-dimensionmemory, in accordance with aspects of the present technology, is shown.In one implementation the array can be a 4-dimension, 3-dimension or2-dimension weight array and the 2-dimension memory can be a memorymacro. The method of fitting the array into a 2-dimension memory will beexplained with reference to FIGS. 46-52 . The method can includeexpanding the dimension of a 3-dimension or a 2-dimension array, at4510. If the array is a 3-dimension array of kernel width (S), a kernelheight © and input channels ©, the array can be expanded to a4-dimension array of kernel width (S), a kernel height ©, one inputchannel and output channels ©, as illustrated in FIG. 46 . If the arrayis a 2-dimension array of input channels © and output channels (M), thearray can be expanded to a 4-dimension array of a single kernel width, asingle kernel height, input channels © and output channels (M), asillustrated in FIG. 47 .

At 4520, the 4-dimension array, expanded 3-dimension array or expanded2-dimension array can be quantized, as illustrated in FIG. 48 . Eacharray element can be quantized to an 8-bit value. Each filter can alsoinclude a single bias value (b) 4810, 4820 and one scaling exponent(exp) 4830. The single bias value 4810, 4820 can comprise two elemententries. In one implementation, the single bias value 4810, 4820 can beencoded as a Bfloat16 value.

At 4530, the filters of the quantized array can be unrolled and the biasvalue and scaling exponent can be appended, as illustrate in FIG. 49 .In one implementation, corresponding entries from each channel can besequentially arranged after the bias value 4810, 4820, and the scalingexponent can be added at the end to produce M flattened output channels.The M flattened output channels can be characterized by length R×S×C+3.Each M flattened output channel corresponds to a virtual channelcharacterized by a virtual channel height (vch) of R×S×C+3.

At 4540, the unrolled and appended filters can be reshaped to fit into aphysical channel of a memory, as illustrated in FIG. 50 . The reshapedfilters can be characterized by a weight prefetch height and an entriesper virtual channel width. The reshaped filters can be padded with zeroelement values if necessary to fit the physical channel of the memory.In one implementation, the physical channel of the memory can be thephysical channel of a memory macro.

At 4550, the reshaped filters can be rotated, as illustrated in FIG. 51. The rotated filters can comprise M virtual channels (e.g., outputfilters). At 4560, virtual channels of the rotated filters can be packedphysical channels of the memory, as illustrated in FIG. 52 . The Mvirtual channels of the rotated filters can be sequentially stored inthe plurality of physical channels of the memory. Physical channels ofthe memory can be padded with zero (0) values if necessary, such that aweight array for a new layer starts at a first physical channel boundaryof the memory.

Again, organizing each of the memory regions into a plurality of memoryblocks and coupling a core group of a respective processing region to aset of memory blocks that are proximate to the given core group, whilenot coupled to memory blocks in the adjacent memory regions that aredistal from the given core group, can increase memory bandwidththroughput. Providing more, but smaller, flat memory bocks by organizingeach of the plurality of memory regions into respective set of aplurality of memory blocks can provide increased memory bandwidth forincreased performance. Further increasing the number of memory blocks ineach of the plurality of first memory regions can further increase thememory bandwidth. Referring to FIGS. 53A-53D, organization of each ofthe plurality of first memory regions into a plurality of columns androws, in accordance with aspects of the present technology, is shown.Each memory region 5310 can be organized into a plurality of memoryblocks of m blocks wide and n blocks long, wherein m and n can bedifferent or equal. In one implementation, memory regions 110-130,202-210, 302-310, 402-408, as described above with reference to FIGS.1-5 , can be between 2 and 128 channels wide. In another implementation,the memory regions 110-130, 202-210, 302-310, 402-408 can be between 2and 128 words wide. A fetch/write back unit can fetch sets of memoryblocks from an adjacent one of the plurality of first memory regions andwrite back sets of memory blocks to another adjacent one of the firstmemory regions in accordance with a dataflow configuration. For example,a fetch unit of a respective compute core can be configured to fetchfrom a set of memory blocks of a respective adjacent one of theplurality of first memory regions. In one implementation, the set ofmemory blocks can correspond to a channel width of the compute core orcache width of the fetch unit as illustrated in FIG. 53A. Additionaldata from sets of memory blocks can then be fetched into the cache ofthe fetch unit, as illustrated in FIGS. 53B-53D respectively. Similarly,a write back unit of a respective compute core can be configured towrite data back to sets of memory blocks of respective adjacent one ofthe plurality of first memory regions, as illustrated in FIG. 54A. Forexample, data can be written back to a first set of memory block of arespective adjacent one of the plurality of memory regions. Additionaldata can then be written back to the next set of memory blocks of therespective adjacent one of the plurality of memory regions. The wideplurality of first memory regions organized into a plurality of columnsand rows, in accordance with aspects of the present technology,advantageously reduces the number of memory access cycles, which cansmother the pipeline, improve arbitration and better latency hiding.However, some compute functions, such as reshape, may need to be basedon multiples of the memory block line width.

Referring again to FIGS. 3-5, 10 and 11 , the compute cores can beconfigured to compute functions including, but are not limited to,vector products, matrix-dot-products, convolutions, min/max pooling,averaging, scaling, and or the like. For example, near memory (M)compute cores can compute up-sampling, deconvolution, separableconvolution, pointwise convolution, MP convolution, and the likefunctions. The arithmetic (A) compute cores can be configured to computemaximum, minimum, subtract, multiply, concatenate, sigmoid/logisticactivation, hyperbolic tangent, mish, swish, constant add, constantmultiply, clip and the like functions. In yet another example, rescaleor the like functions can be supported by a graph processing core.

Compute functions such as the reshape function can be implemented by thecontrol circuitry and or inter-layer communication (ILC) units 450-456.Reshaping can be supported by adjusting corresponding increment anddecrement counts of the inter-layer communication unit. For example, theincrement count can be set to +4 and the decrement count can be set to−6 to reshape a 6×4 producer output to a 4×6 consumer input in a per rowILC synchronization scheme, as illustrated in FIG. 55 . A deconvolution,also known as a two-dimension transpose convolution (Conv2Dtranspose),can include kernel transformation and up-sampling as illustrated inFIGS. 56A-56D. For example, the Conv2Dtranspose can be implementing bytransposing (e.g., flipping) kernel weights, while the strides happen onthe output feature map instead of the input. This is equivalent toinserting an up-sampling layer with inserted zeros before thetwo-dimension convolution with transposed kernel weights, as illustratedin FIG. 57 . A sigmoid function is defined as:

$\begin{matrix}{{S(X)} = \frac{1}{1 + e^{- x}}} & \end{matrix}$

The sigmoid function can be approximated in the compute cores using thepiecewise equation:

$\begin{matrix}{{A(X)} = \left\{ \begin{matrix}{{0\ x} \leq {- 2}} \\{{\frac{x}{4} + \frac{1}{2}\  - 2} < x < 2} \\{{1\ x} \geq 2}\end{matrix} \right.} & \end{matrix}$

as illustrated in FIG. 58 .

Generally, feature maps can be encoded as integer data, B-float data,group B-float or the like. Referring now to FIG. 59 , a feature map ofkernel width (X), a kernel height (Z) and channels (Z), in accordancewith aspects of the present technology, is shown. For integer data,feature map pixels can be encoded as n-bit integer values. For example,the feature map pixels can be represented as 8-bit integers. Thefixed-point location of the integer can be estimated offline using apilot data set. The pilot data set can be utilized to encode the data sothat the entries share the same exponent (e.g., static exponent).However, if the runtime conditions or data set differs from the pilotdata set, the effective precision is significantly degraded, and networkbranching can be difficult.

In another implementation, the feature map pixels can be encoded asBrian Floating Point (B-float) values, including a base and exponent.For example, the feature map pixels can be represented by 16 bits,including an 8 bit signed fraction and 8 bit exponent. The 8 bit signedfraction can include a sign bit, 7 explicitly stored faction bits and 1hidden fraction bit, as illustrated in FIG. 60 . Each B-float encodedentry can have its own dynamic exponent. The B-float encodingadvantageously does not need a pilot data phase, and advantageouslyadapts to runtime conditions. However, B-float encoded data utilizesdouble the memory storage and memory bandwidth as compared to integerencoded feature map data.

In yet another implementation, the feature map pixels can be representedby as B-float values, wherein each group of n-channels of pixels havetheir own dynamic exponent. The n-channels should be less than or equalto the number of physical channels. B-float encoding, wherein groups ofn-channels are encoded with a given dynamic exponent is referred toherein as Group B-float encoding. Group B-float encoding advantageouslydoes not need a pilot data phase, and advantageously adapts to runtimeconditions. In most cases, Group B-float encoding can advantageouslyutilize substantially the same memory storage and memory bandwidth asinteger encoded data, by storing the group B-float encoded data, inaccordance with aspects of the present technology.

Referring now to FIG. 61 , a logical view of a feature map encoded inGroup B-float from an output side of a computer core, in accordance withaspects of the present technology, is illustrated. For a layer ‘i’, theoutput channels 0-8 can be encoded by 8-bit exponents e, and 8 bitfractions ‘m.’ The exponent is across multiple channels, since eachoutput channel has one output entry per pixel. Referring now to FIG. 62a logical view of a feature map encoded in Group B-float from an inputside of a compute core, in accordance with aspects of the presenttechnology, is illustrated. For a layer ‘i+1’, the exponent of thefeature map 6210 are shared along a same axis as weights per-channelquantization 6220. Referring now to FIG. 63 , another logical view of afeature map and weights encoded in Group B-float from an input side of acompute core, in accordance with aspects of the present technology, isillustrated. Referring now to FIG. 64 , storage of B-float encodedfeature map data in a narrow flat memory organization, in accordancewith aspects of the present technology, is shown. In the narrow flatfirst memory organization, a base and exponent of B-float encoded valuefor feature map pixels can be stored in corresponding word lines.Referring now to 65, storage of B-float encoded feature map data in awide memory organization, in accordance with aspects of the presenttechnology, is illustrated. In the wide memory organization, a base andexponent of B-float encoded value for each of a plurality of feature mappixels can be stored in corresponding word lines. For B-float encoding,each pixel entry can have its own dynamic exponent. Therefore, theexponent for each pixel entry needs to be stored with the respectivebase. Referring now to FIG. 66 , storage of Group B-float encodedfeature map data in a wide memory organization, in accordance withaspects of the present technology, is illustrated. In the wide memoryorganization, an exponent is the same for the pixels of a group ofchannels. Therefore, the base of the pixels value can be stored with oneinstance of the dynamic exponent for the given group of channels.Referring now to FIG. 67 , accuracy of calculations on Group B-floatencoded ResNet-50 feature map pixels values for different group sizes isillustrated. Referring now to FIG. 68 , accuracy of calculations onGroup B-float MobileNet feature map pixels values for different groupsizes is illustrated. The combination of Group B-float encoding and widememory organization for use in memory regions 110-130, 202-210, 302-310,402-408, as described above with reference to FIGS. 1-5 , canadvantageously provide almost two times (2×) the memory bandwidth forthe same memory width. The combination of Group B-float encoding andwide memory organization can also advantageously reduce the on-chipmemory storage need by almost one half (½). The accuracy achievable forcomputations utilizing Group B-float encoded, including but not limitedto neural network computations, can be substantially equal to B-floatencoded values.

The foregoing descriptions of specific embodiments of the presenttechnology have been presented for purposes of illustration anddescription. They are not intended to be exhaustive or to limit thepresent technology to the precise forms disclosed, and obviously manymodifications and variations are possible in light of the aboveteaching. The embodiments were chosen and described in order to bestexplain the principles of the present technology and its practicalapplication, to thereby enable others skilled in the art to best utilizethe present technology and various embodiments with variousmodifications as are suited to the particular use contemplated. It isintended that the scope of the invention be defined by the claimsappended hereto and their equivalents.

What is claimed is:
 1. A memory processing unit (MPU) comprising: afirst memory including a plurality of memory regions, wherein one ormore of the plurality of memory regions are configured in acorresponding pluralities of memory blocks, wherein the memory blocksare configured to store Brian Floating Point (B-float) encoded data; anda plurality of processing regions interleaved between the plurality ofregions of the first memory, wherein the processing regions include aplurality of core groups, wherein the core groups include one or morecompute cores.
 2. The MPU of claim 1, wherein the memory blocks arefurther configured to store Group B-float encoded data.
 3. The MPU ofclaim 2, wherein the Group B-float encoded data comprises Group B-floatencoded feature map pixels values.
 4. The MPU of claim 2, wherein theplurality of memory blocks of each of the plurality of regions of thefirst memory are arranged in a plurality of columns and rows.
 5. The MPUof claim 4, wherein a plurality of bases and an instance of a givenexponent for a corresponding group of channels of the Group B-floatencoded data are store in a corresponding row of memory blocks of acorresponding region of the first memory.
 6. The MPU of claim 5, whereinthe exponent for corresponding groups of channels of the Group B-floatencoded data are dynamic.
 7. The MPU of claim 1, wherein one or more ofthe plurality of core groups of a respective one of the plurality ofprocessing regions are coupled between adjacent ones of the plurality ofmemory regions of the first memory, and between adjacent core groups ofthe respective one of the plurality of processing regions.
 8. The MPU ofclaim 1, wherein the plurality of core groups of respective ones of theplurality of processing regions are coupled between adjacent ones of theplurality of memory regions of the first memory.
 9. The MPU of claim 1,wherein compute cores of respective ones of the core groups areconfigured in one or more compute clusters, wherein compute cores in agiven compute cluster are configured to compute a given computefunction.
 10. The MPU of claim 9, wherein one or more compute groupsinclude one or more memory M-cores and one or more arithmetic A-Cores.11. A memory processing unit (MPU) comprising: a first memory includinga plurality of memory regions, wherein the plurality of memory regionsare configured in corresponding pluralities of memory blocks, andwherein the memory blocks are configured to store Group B-float encodedfeature map pixels; and a plurality of processing regions columnalinterleaved between the plurality of regions of the first memory,wherein the plurality of core groups of respective ones of the pluralityof processing regions are coupled between adjacent ones of the pluralityof memory regions of the first memory and between adjacent core groupswithin the respective processing region.
 12. The MPU of claim 11,wherein the plurality of memory blocks of each of the plurality ofregions of the first memory are arranged in a plurality of columns androws.
 13. The MPU of claim 11, wherein a plurality of bases and aninstance of a give exponent for a corresponding group of channels of theGroup B-float encoded data are stored in a corresponding row of memoryblocks of a corresponding region of the first memory
 14. The MPU ofclaim 10, further comprising one or more memory regions of a secondmemory coupled to the plurality of processing regions.
 15. The MPU ofclaim 14, wherein the second memory is configured to store weightvalues.
 16. The MPU of claim 14, wherein respect ones of the secondmemory regions are coupled to respective ones of the plurality ofprocessing regions.
 17. The MPU of claim 14, further wherein the computecores in corresponding core groups of the plurality of processingregions are: configurable for core-to-core dataflow between adjacentcompute groups in respective ones of the plurality of processing regionsthrough one or more corresponding memory blocks of a correspondingmemory region; configurable for memory-to-core dataflow from respectiveones of memory blocks of the plurality of regions of the first memory toone or more cores within adjacent ones of core groups of the pluralityof processing regions; configurable for core-to-memory dataflow from oneor more cores within ones of the plurality of core groups of theplurality of processing regions to adjacent ones of the memory blocks ofthe plurality of regions of the first memory; and configurable formemory-to-core dataflow from the second memory region to one or morecore groups of corresponding ones of the plurality of processingregions.
 18. The MPU of claim 11, wherein: the first memory comprises astatic volatile memory; and the second memory comprises a non-volatilememory.
 19. A memory processing method comprising: configuring a firstmemory to store Group B-float encoded data, wherein the first memoryincludes a plurality of regions; configuring data flow between computecores of one or more of a plurality of processing regions andcorresponding adjacent ones of the plurality of regions of the firstmemory; configuring data flow between a second memory and the computecores of the one or more of the plurality of processing regions;configuring data flow between compute cores within respective ones ofthe one or more of the plurality of processing regions; configuring oneor more sets of compute cores of one or more of the plurality ofprocessing regions to perform respective compute functions of a neuralnetwork model; loading weights for the neural network model into thesecond memory; loading activation data for the neural network model intoone or more of the plurality of regions of the first memory;synchronizing data movement between one or more compute cores producinggiven data and one or more other compute cores consuming the given databased on the neural network model.
 20. The memory processing methodaccording to claim 19, wherein the plurality of regions of the firstmemory each include a plurality of memory block arranged in a pluralityof columns and rows.
 21. The memory processing method according to claim20, further comprising configuring the first memory to store a pluralityof bases and an instance of a given exponent for a corresponding groupof channels of the Group B-float encoded data in a corresponding row ofmemory blocks of a corresponding region of the first memory.
 22. Thememory processing method according to claim 21, wherein the exponent forcorresponding groups of channels of the Group B-float encoded data aredynamic.